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ABSTRACT 
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Chapter 49 

Assessing the Quality of Online 
Instruction 

Integrating Instructional Quality and Web 
Usability Assessments 

Anthony Ciavarelli 



A great deal has been written in the educational literature about 
the use of distance education, given the rapid expansion of the World 
Wide Web on the Internet. Educators and trainers are enthusiastic about 
the potential for reaching learners across a medium that allows 
“anywhere and anytime” teaching. Web-based courses are now 
available or under development at many academic institutions and 
through corporate universities. Such online courses allow individuals 
to complete needed educational or training programs from their desktop 
computers at a convenient tune and place. Online courses are especially 
desirable for learners who are unable to attend traditional classes on 
campus because of their remote geographic location or their limited 
time availability. 

Educators are concerned, however, about ensuring the quality 
of online courses, and many question whether online courses can 
maintain the same high standards of excellence as traditional classroom 
instruction. One way to ensure quality control over online instruction 
is to establish an ongoing instructional quality assessment process. In 
this chapter I discuss background information and several 
considerations for assessing the quality of online instruction. 

Questions Educators Ask About Online Instruction 

Administrators, educators, and students question several key 
issues of online instructional quality, and some question the feasibility 
of attaining critical learning objectives outside the traditional classroom 
and laboratory. Among commonly asked questions are the following: 
• Can complex learning objectives associated with problem 
solving and critical thinking skills be taught online? 
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• How does one address the issues of socialization and 
collaboration in learning among students using online 
instructional formats? 

• What makes an online course successful or unsuccessful? 

® How can we ensure acceptable quality of online instruction? 

• How can we demonstrate that students receiving online 
instruction have equitable standards, attention, and resources 
compared to on-campus students? 

• What kinds of data should be collected to measure the quality 
and the effectiveness of online education? 

In this chapter I discuss the background and foundation of a 
proposed conceptual framework for assessing the quality of online 
instruction. This e-Leaming Assessment Framework is presented in 
the chapter appendix. In this framework, I consider factors related to 
human learning and motivation, instructional quality standards, best 
web teaching practices, and web page design and usability guidelines. 
My premise is that the quality of web-based instruction is a result of all 
these factors. 

The development of high-quality web learning environments 
requires careful planning and a systematic development process. The 
development process needs to take into account what we know about 
human learning capabilities and human motivation to learn. High-quality 
online instruction also depends on careful application of established 
principles of instructional design, use of the best web design practices, 
good human engineering, and adequate provisions for institutional 
support. The background and formation of this assessment framework 
is discussed below. 



Instructional Design Issues 

Instructional design encompasses several issues, including 
pedagogy, development of instructional systems, theories of human 
learning, and how web-based instruction must differ from traditional 
instruction. All these topics are covered in this section. 

Pedagogy 

The growth of online instruction has raised new interest and much 
discussion about pedagogy, or how best to teach. The term actually 
comes from Greek roots, paid and agogos, and in translation means 
“the art of teaching children.” The primary audience for much of our 
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existing online instruction, however, is adults seeking education in a 
venue that will allow them to continue their employment while working 
on educational goals part time. Adults have different needs than children. 
As Knowles (1980) so aptly points out, children have little experience, 
are able to focus on academics, are subject centered, and are easily 
motivated by external rewards. Adults, on the other hand, have greater 
experience levels, tend to be focused on acquiring job skills, are more 
problem centered, and are self-motivated to learn. A good teacher knows 
that individual learners have different levels of experience, capabilities, 
and motivation levels, and takes such factors into consideration in his 
or her approach to teaching. 

I have encountered people involved with online instruction who 
use pedagogy in place of a particular technology or course-delivery 
method. They might say, for example, that the use of online instructional 
formats, such as online discussion forums or chat rooms, represent a 
new pedagogy. In my view, pedagogy refers, or should refer, to the art 
of effective teaching, not to a specific technology or content-delivery 
method. Whenever we plan to teach, on the web or in the classroom, 
we should first address the pedagogical issues of how best to teach our 
subject to the expected audience. Some questions that every teacher 
should ask when developing a course are these: 

® Who am I trying to teach? 

° What content do I need to teach? 

® How can I best organize this content? 

® What is the best presentation strategy and lesson sequence? 

® What is the best way to deliver the content? 

• How will I know when I have succeeded in teaching what I 
intended to teach? 

Professional instructional designers typically ask themselves these 
or similar questions when they apply a systems approach, sometimes 
referred to as the instructional systems development (ISD) process to 
course development. 

Instructional Systems Development 

ISD is a process model that defines steps or tasks that course 
developers need to complete to ensure that they pay adequate attention 
to critical components of instruction. The ADDIE model — named after 
the five critical process steps or phases of analysis, design, development, 
implementation, and evaluation — is thoroughly described by Hodell 
(2000) and will not be discussed in any detail here. 
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The importance of the ADDEE model to assessment is twofold. 
First, the model has enjoyed considerable success in the training 
community as a means to standardize instruction and impose quality 
standards. An ADDEE approach has helped to improve some of the 
more poorly plaimed and haphazard training interventions. In traditional 
classroom education, there has been little acceptance and application 
of this kind of systematic approach. In contrast, the application of ISD 
methods to online instmction is receiving much attention, probably 
because efforts to convert classroom content to web delivery require 
careful planning, design, and development in order to achieve a seamless 
integration of course content and teaching strategies with course- 
delivery technologies. 

The ISD model and some activities typically engaged in at each 
of the development phases are briefly summarized here (based on 
Hodell, 2000, pp. 12-13): 

Analysis. Activities include assessment of student learning needs, data 
gathering regarding subject matter or content, and organizational and 
technology implementation issues. 

Design. The focus is on specification of learning objectives and 
performance standards, defining subject matter content, defining the 
curriculum, designing lesson plans and tests, selecting media, and 
addressing resource and support needs. 

Development. This stage involves production of instmctional materials 
and student and teacher guides, courseware authoring, and software 
development, according to specifications in the design. 

Implementation. Implementation addresses delivery, management, and 
control of the education and learning process. 

Evaluation. Continuous measurement and evaluation of the 
instructional products and processes takes place in all phases of ISD. 

One goal of applying the ADDIE model is to ensure that all critical 
components of instruction, such as inclusion of learning objectives, 
appropriate teaching strategies, and relevant assessment methods, are 
present in any resulting course or curriculum. Most educators know 
that the mere application of a theory or process cannot guarantee high- 
quality instruction will result. Educators’ opinions vary, however, as to 
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what teachers must do to ensure that learning takes place as intended. 
Some of the differences in opinion about learning may trace their roots 
to varying views about human learning. Ultimately, the student who 
experiences the result of such a process will be the best person to judge 
the quality of instruction. The perception of quality will be determined 
by the learner’s own goals and expectations and his or her unique 
experiences taking the course. 

Views About Human Learning 

There are many views of human learning, but perhaps the most 
common and relevant are behavioral, cognitive, and constructive 
learning theories. I briefly compare and contrast each of these viewpoints 
here. 



Behavioral Learning Theory 

This approach to teaching humans actually evolved from studies 
of animal behavior in the psychology laboratory. Early behaviorists 
(e.g., John Watson, Edward Thorndike, and B. F. Skinner) studied only 
learning that could be easily observed and measured. As a result their 
learning approaches tended to focus on tasks and skills that could be 
objectively defined and taught through hands-on practice with feedback 
on performance results. Techniques such as defining behaviors to be 
taught; arranging situations so that the behaviors could be attempted; 
and providing reinforcement, or feedback about performance, are still 
used today in many training environments. Such an approach may work 
well in an application such as training equipment operators to memorize 
and practice specific operating procedures. But a purely behavioral 
training method is difficult to apply to teaching complex human problem 
solving and critical thinking skills (Ford, 2000). 

Bell (1985, pp. 36-40) nicely summarized a behavioral approach 

to education and training: 

1. Define learning objectives as measurable outcomes. 

2. Define the learning prerequisites or entry skill level. 

3. Present instruction in a progressive sequence of least to 
most difficult. 

4. Make connections between current and previous instruction. 

5. Present examples, sample problems, and guidance for 
problem solving. 

6. Teach basics, give a variety of examples, give practice with 

feedback, and test performance at the end of instruction. 
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7. Provide immediate feedback on results. 

8. Provide information about how to correct errors (remedial 
instruction). 

9. Redirect the student’s instruction based upon diagnosis of 
performance problems. 

Cognitive Learning Theory 

Cognitive theorists recognize that people are different from 
animals in their thinking abilities, and they are capable of guiding their 
own learning processes based upon abilities, or cognitive structures, 
that they establish through a lifetime of learning. Furthermore, cognitive 
psychologists maintain that humans are both extrinsically and 
intrinsically motivated to learn. They do not depend solely upon external 
events to control their learning and motivation to learn. Thus, cognitive 
learning theorists suggest that educators take advantage of these thinking 
processes, self-organizing learning strategies, and intrinsic motivation 
to learn (Ford, 2000; Johnson & Thomas, 1994). 

Merrill and his associates developed a theory of instruction based 
upon cognitive science principles (Merrill, 1983; Merrill, Reigeluth, & 
Faust, 1979) . This approach to instruction is sometimes called a 
prescriptive approach to instruction, or prescriptive instructional theory, 
because the strategies for delivering and assessing instruction are based 
upon specific learning objectives and their associated performance 
requirements. The prescriptive approach to instruction follows a defined 
process to ensure that certain critical components for producing high- 
quality instruction are present. The basic components of Merrill’s 
prescriptive instructional design theory, sometimes referred to as 
component display theory or component design theory (CDT), are as 
follows (Ciavarelli, 1988, p. 13): 

1. Analyze and classify the task or activities the learner is to 
perform (as specified in learning objectives). 

2. Select the most appropriate teaching strategy based upon 

' the type of learning and subject matter content. 

3 . Evaluate instructional quality based upon the adequacy and 
consistency of the key components of instruction across 
learning objectives, teaching strategies, and assessment 
methods. 

The CDT approach actually borrows heavily from behavioral 
methods in that it incorporates a systematic process of writing learning 
objectives as learning outcomes. The instructional presentations are 
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highly structured bused upon the particular classification of learning 
objectives and instructional content. Certain elements of cognitive 
psychology are added, however, to accommodate concept learning and 
problem-solving tasks; for example, learning strategies that support 
the learning encoding process. These cognitive learning strategies 
include such things as the use of mnemonics, mental rehearsal, and 
subject matter organizers, highlighting, and isolation of core learning 
materials to draw the learner’s attention to key information. Another 
fundamental tenet of this approach is to allow some means for the learner 
to control the learning process. 

For example, in some forms of computer-based instruction, learner 
control is achieved by incorporating learner- selected options or choices 
(Merrill, 1980). The learner is allowed to select the desired level of 
difficulty and also can access additional instructional support and help. 
The instruction typically includes a broad range of subject matter 
explanations, illustrations, and examples that are helpful in teaching 
specific concepts and principles. For example, when CDT is used to 
design a concept lesson, the instructional sequence might go something 

like this (Ciavarelli, 1988, p. 14): 

1 . Define the concept and highlight or isolate the key attributes 

that characterize or typify it. 

2. Provide examples and instances that represent the concept. 
Include both examples and nonexamples (examples that 
do not represent the concept). 

3. Allow the learner to apply the definition of the concept 
and to list the attributes needed to classify various instances 
or examples of concept classes. 

4. Assess learning by having the student classify new instances 
or examples as examples or nonexamples of the concept. 

Merrill (1994) refined his CDT to provide a broader focus on 
course structuring and to accommodate the use of expert systems 
technology (artificial intelligence) in providing advisory tutoring to 
students taking computer-based or online courses. Using such “expert 
tutor” technology, the best advice of a teacher can be incorporated into 
the instruction to guide and advise the student along different learning 
paths, and to help correct errors and possible misconceptions. 

Constructive Learning Theory 

Constructive learning theory suggests that students do not just 
passively receive education; they actively engage in organizing and 

9 



Assessing the Quality 



702 



making sense of the information they receive. In essence, they construct 
their own knowledge base by integrating new information and 
experiences with pre-existing (already learned) information. According 
to constructive learning theorists, learning is best achieved by setting a 
context in which students can readily understand where new knowledge 
fits into their own experience and how such knowledge can be used in 
a real-world setting. The teaching focus is on establishing a learning 
environment in which the learner is involved in actively constructing 
knowledge from the ongoing learning experience and in connecting 
new knowledge to previous experiences (Kerka, 1997). 

Teachers play an important role in the learning process by 
arranging instruction that encourages students to engage in active 
learning situations and participate in problem-solving activities, 
sometimes working alone and other times working in groups. The 
teacher helps to organize problem-centered activities, then guides and 
encourages inquiry and exploration of possible solutions. Instructional 
materials, problem-solving situations, and assessments are often set in 
practical contexts that invite students’ interest because the learning 
experience is directly applicable to real-world environments. This aspect 
of constructive learning theory is sometimes referred to as situated 
learning, in which knowledge is formed and made more me anin gful 
by establishing an authentic learning situation. (Kerka, 1997). 

The most effective learning, then, takes place when learners 
attempt to understand and make sense of the learning experience and 
begin to fit their new knowledge into their own unique experience base 
or conceptual framework. From the constmctivist point of view, the 
most effective learning takes place when teachers allow students to 
work with engaging, problem-centered instructional materials; when 
they provide guidance along the way as needed; and when they 
encourage collaboration among learners. 

People learn in social contexts by observing others and by actively 
participating in conversation and functions with others (Bandura, 1977). 
The acquisition of language and culture plays a large role in learning 
higher cognitive skills and in building motivation and strategies for 
learning how to learn (Vygotsky, 1978). 

Bruner (1990), who coined the term discovery learning, believes 
that human learning is inextricably embedded in an individual’s culture. 
Humans learn because they want to make sense of their world and 
understand its workings. Knowledge about the world, then, is a social 
construction composed of the information that learners have acquired 
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within a cultural or social framework. 

An obvious advantage of web instruction is that one can reach 
students at a distance, anytime and anywhere. From a constructivist 
standpoint McManus (2001) points out that another key advantage is 
that the web represents a vast source of information resources that can 
be used in an instructional program. Teachers can, in the course of 
teaching on the web, connect their students with noteworthy institutions 
of learning, libraries, and museum websites to provide motivation and 
enhance the educational experience. 

The availability of vast web resources invites a constructivist 
approach to teaching. Spiro, Feltovich, Jacobson, and Coulson (1991) 
suggest that higher-order learning is not taught very effectively with a 
behavioral objective approach in which the educator specifies learning 
outcomes and controls the learning process. From the constructivist 
view, it should be the learner who sets the learning goals and directs 
the knowledge acquisition process. Spiro and associates believe that 
multiple interconnections among knowledge components should be 
emphasized. In addition, the web provides for a high level of 
interconnections among knowledge sources (McManus, 2001). The 
teaching strategy is based upon providing guidance to the student by 
arranging student-directed web-leaming activities that include the use 
of web resources, and by facilitating collaboration among students via 
Internet communication modes, such as e-mail and discussion forums 
or chat rooms. 

Web instruction lends itself to constructive forms of instruction 
through planning engaging learning activities and through planning 
collaborative learning projects (using communication features). 
Encouraging collaborative learning and intercommunication among 
students helps to build a learning community and to offset feelings of 
isolation that students who take technology-based educational programs 
may experience. Sandra Kerka (1996) suggests a number of ways to 
improve online instruction; 

(1) Understand the technologies’ strengths and weaknesses; 

(2) provide technical training and orientation; (3) plan for 
technical failures and ensure access to technical support; (4) 
foster learning to learn, self-directed learning, and critical 
reflection skills; (5) develop information management skills 
to assist learners in selection and critical assessment options; 

(6) structure learner-centered activities for both independent 
and group work that foster interaction, (p. 3) 
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Constructivism and Authentic Assessment 



The constructivist movement places great emphasis on putting 
the learner in control of the educational process. Going hand in hand 
with this shift in focus is the movement toward authentic assessment, 
in which teachers examine students’ performance on meaningful 
intellectual tasks (Wiggins, 1990). Wiggins (1990, p. 1) raises the 
following key points: 

• Traditional classroom exams do not transfer well to the real 
world, making them poor substitutes for measuring student 
performance in relation to realistic settings. 

• Authentic tests attempt to develop engaging student tasks 
that simulate realistic tasks such as conducting research, 
collaborating with other students on a research paper or 
project, or reporting on current social events or scientific 
breakthroughs. 

• Authentic tests are representative of ill- structured problem 
domains and are thought better to prepare a student for 
professional practice and transfer of education to real-life 
situations. 

Authentic testing is often associated with the development of 
assessment rubrics. A rubric is an assessment tool that corresponds as 
closely as possible to a real-world problem or situation (Moskal, 2000). 
This form of assessment is considered much more appropriate to 
enhancing the learning experience than traditional, objectively scored 
classroom exams. 

Recent Ideas in Instructional Design 

Sonwalker (2001a) argues that web instruction, with some of its 
new multimedia enhancements (video, animation, and simulation), 
requires a new pedagogy. He describes four applicable learning styles: 
incidental (event-driven reactive learning), inductive (the introduction 
of instances or examples of a concept to exemplify a principle being 
taught), deductive (presentation of simulations or graphic or 
mathematical representations that illustrate trends and drive learners 
to draw conclusions), and discovery (learning within the knowledge 
domain via self-directed inquiry). Sonwalker suggests multiple modes 
of presentation to allow for different and wide-ranging learning 
pathways. A very clever three-dimensional pedagogical learning cube 
is proposed as a teaching strategy model or conceptual framework. 
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The cube, similar in appearance to a Rubik’s Cube puzzle, depicts an 
instructional system comprising the teacher, student, learning style, and 
media type. The model can be used to “organize and sequence 
multimedia content assets in a pedagogically distinctive learning path 
that matches the style of the individual learner” (Sonwalker, 2001b, p. 
12 ). 




Web Page Design and Usability Guidelines 

The part of a computer system (including the Internet and various 
websites) that the user sees and communicates with is called the user 
interface. The user interface provides the means for the user to interact 
with the computer. Anyone who has attempted and failed to program a 
videocassette recorder correctly, or who has accidentally deleted or 
lost a computer file, knows the frustration of interfaces that are poorly 
designed for the typical user. Web-based instruction is embedded in a 
computer system that, if not carefully designed, may result in the same 
kinds of operational difficulties and consequent user frustration. 

Over the years, human factors engineers specializing in human- 
computer interaction (HCI) and usability engineering have attempted 
to influence the design of computing systems to make them easier to 
use. HCI is a discipline concerned with the design, evaluation, and 
implementation of interactive computing systems. By applying HCI 
design principles, developers of computer systems have learned to 
design systems more carefully to ensure that computers are easier to 
use and less error prone. HCI concerns itself with involving the user in 
the design process, applying design guidelines to simplify system 
operation, and providing the user with assistance when needed. An HCI 
designer works mainly to improve user interface designs. One important 
part of the human factors method is to conduct usability testing 
throughout development. Usability refers to a measure of the quality 
of the user’s experience when interacting with the system such as a 
personal computer, videocassette recorder, or website (Stefanyshyn, 
2001). Usability testing is conducted to test the ease of learning and 
simplicity of operation of the interface by having a sample of intended 
users try out the system. Usabihty engineers may observe users operating 
the system and take note of difficulties, or they may conduct surveys 
and interviews to obtain data regarding the ease of operation. Based on 
such information, the usability engineer may redesign the human 
interface to make it easier to use (Jordan, 1998). One does not have to 
be an engineer to consider human factors in web page designs for online 
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instruction, however. Some key design questions regarding the human 
interface are these (Jones, 1989, p. 13): 

• How should the function of the system be described and 
presented to the user? 

• How can the design of the user interface help the user to 
understand and use the system? 



I would add one more question for the user who becomes totally 
lost in the system: How do I get out of this mess? The goal of HCI and 
usability engineering is to design a system that is easy to operate, 
intuitive, and quickly learned. The following guidelines are used in 
meeting such design objectives (Jones, 1989, pp. 21-45): 

• Maintain consistency in display format, information layout, 
and position. 

• Use landmarks and signposts to show user his or her present 
(web page) location, the path traversed, and what lies ahead. 

• Indicate the present condition or state of the system through 
operating messages such as “downloading now,” “please 
wait,” “estimated time to complete,” and the like. 

• Indicate the start and completion of each task. 

• Give user a way out of a mess. For example, give the user a 
way to go to the home page and start over. 

In summary, the user interface is designed to maintain consistency 
in information formats and location of content on a web page. The 
most usable systems pay attention to the target users’ needs and abihties. 
Functional operation and navigation choices are compatible with users’ 
experiences with other similar systems, and the system provides 
feedback to the user, such as acknowledging inputs, advising the user 
of processing wait periods, and providing navigational assistance, error 
alerts, and corresponding recovery methods. The specific application 
of these principles, and other key design issues and usability methods 
in creating user-friendly web pages, are covered extensively in Jakob 
Nielsen’s popular book Designing Web Usability (2000). The book also 
contains many examples of good and bad web pages to illustrate design 
principles. 



Where am I? 

What can I do here? 

How did I get here? 

Where can I go, and how do I get there? 




Assessing the Quality 



707 



er|c 



Institutional Support Issues 

Online courses, as I have mentioned, must be carefully planned, 
designed, and executed. Agencies that govern academic accreditation 
or set quality standards have considered establishing criteria regarding 
institutional responsibilities and requirements in online instruction. I 
have incorporated some of the criteria being considered by the Western 
Association of Schools and Colleges (WASC) and the National 
Education Association (NEA) in the e-Leaming Assessment Framework 
in the appendix. By way of summary, an institution planning to offer 
online courses has to consider where online course offerings fit into its 
academic mission, as well as to plan for adequate resources and facilities 
to provide the needed technology, faculty development, and student 
services support infrastructure. Academic institutions that have a 
successful campus program and are planning to undertake new programs 
of distance learning would be wise to follow the example of the British 
Open University, one of the more prominent and successful distance 
learning institutions in the world. The ingredients of success for the 
Open University, as reviewed by Sir John Daniel (2001), are quite 
revealing: “The Open University is lucky in that it does not have to be 
satisfied with the assumption of quality. Britain now has a ferocious 
quality assessment system that sends groups of peers, under state 
supervision, to judge the quality of teaching of each discipline in each 
university, against six criteria: (1) curriculum design, content, and 
organization, (2) teaching, learning, and assessment, (3) student 
progression and achievement, (4) student support and guidance, (5) 
learning resources, and (6) quality assurance and enhancement” (p. 3). 
The current high standing of the British Open University is a 
consequence of applying these criteria. The quality of its distance 
learning program is maintained because close attention is given to 
producing excellent course materials, maintaining close personal 
academic support for the students, establishing effective logistics 
(having the right materials at the right place at the right time), and 
encouraging active faculty research. 

Using the e-Learning Assessment Framework 

Education and training are all about learning, teaching, and 
ultimately learner performance as defined by the goals of the institution, 
the objectives of the course, and the special interests of the student. 
Assessment should provide an objective and valid means to judge one’s 
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educational and training accomplishments as an educator and 
administrator of learning. Assessment should provide important 
diagnostic feedback to the student to improve learning, to the teacher 
to improve the instructional process, and to the institution to improve 
its curriculum, support services, and infrastructure. 

I am in the process of developing an online assessment system, 
using a web-based questionnaire survey methodology. The e-Leaming 
Assessment Framework in the appendix provides a useful point of 
departure for specifying measurement dimensions and for constructing 
survey items. Selecting from the appendix, for example, one might 
construct a simple checklist to identify important criteria for designing 
or assessing online courses, as shown in the following examples, one a 
checklist approach and the other a Likert-scale approach. 

Checklist Approach (From Designer’s Perspective) 

[ ] Course learning objectives are clearly stated. 

[ ] The instructor’s role is defined. 

[ ] Interactivity is consistent with the learning objectives or intent. 
[ ] Student collaboration is encouraged when appropriate. 

Likert-Scale Approach (From Student’s Perspective) 

The role of the instructor was clearly defined: 

strongly disagree disagree neutral 

agree strongly agree 

The course content often included active learning tasks: 

strongly disagree disagree neutral 

agree strongly agree 

In this manner, a course designer or an instructor can use items in 
the e-Leaming Assessment Framework to build an assessment tool that 
meets his or her particular interests and requirements. The e-Leaming 
Framework simply provides some assurance that the many dimensions 
of instmctional assessment are considered in the evaluation constmction 
process. The framework incorporates behavioral, cognitive, and 
constmctivist views. It is up to the developer of an assessment instmment 
to decide on the selections most appropriate to the purpose of the 
evaluation. Another important point in developing an assessment 
instmment is to evaluate all components of an instmctional system, 
including the quality and value of the instructional content, the 
instmctor’s performance, the instmctional strategy used, the presentation 
method (lecture, seminar, learning activity), the delivery system, the 
appropriateness and reliability of the technology and media, and the 
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institutional support services. 

Appendix A: e-Learning Assessment Framework 



Instructional Quality Measure 
A. Instructional Quality 


Suggested Readings 


1. Learning objectives are consistent with the 
stated purpose of the course. 


Bell, 1985 


2. Learning objectives are clearly stated in terms 
of performance expected and conditions 
required (behavior, conditions, standard). 


Johnson & Thomas, 
1994 


3. Instruction is adequate to meet specified 


Kerka, 1997 


objectives, including 

a. Content completeness 
and relevance 


Knowles, 1980 


b.Content organization and information 


Merrill n.d. 


sequencing and structure 
c. Balance among general facts, concepts. 


Merrill, 1994 


principles, and process steps in 
applying intended knowledge and skills 
d. Inclusion of relevant examples, 
illustrations, and practice exercises 


Merrill, Reigeluth, & 
Faust, 1979 


4. Exam content is based on learning objectives. 

5 . Exam performance requirements are consistent with 
the learning objectives. 

6. Exam items are well constructed (easy to interpret 
and not ambiguous). 

7. Instructions for taking exams are clear, and 
responses expected on exams are well defined. 

8. Learning strategies are selected to enhance learning. 

9. Instructional content is presented in the most 
effective and efficient way, based upon learning 
strategies selected. 

10. Instructional presentations adequately prepare 
students to perform as specified in learning 
objectives. 

11. Students are given an opportunity to practice and 
review (with feedback and remedial work if needed) 
prior to taking an exam. 

12. Instruction includes learning activities with 
authentic and engaging instructional materials and 
(if applicable) work that requires student interaction 
and collaboration. 

13. Students are given clear instructions and 
mechanisms for obtaining learning assistance and 
help from the instructor in understanding course 
requirements. 


Montague, Willis, & 
Faust, 1980 
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Appendix B. Web Instruction Best Practices 

1 . The role of the instructor is well defined. 

2. The means of access to the instructor are clearly 
indicated. 

3. Communication methods (e-mail, bulletin board or 
forum, chat room) are identified. 

4. Online help with communication methods is 
available to students. 

5. Multiple varieties of interaction are incorporated 
(student learning material, student-student 
interactions, and student-teacher interactions). 

6. Media used are appropriate for learning objectives 
(i.e., media add value to instruction). 

7. Student collaboration is encouraged when 
appropriate. 

8. Content engages students in active learning tasks. 

9. Internet links are meaningfully related to learning 
objectives. 

10. Internet links are appropriately placed so as not to 
interrupt the logical flow of instruction. 

1 1 . Group activities are used to promote learning. 

12. Group activities are used to promote socialization 
and development of a learning community. 

13. Course structure capitalizes on the availability of 
resources on the Internet. 

14. Course structure capitalizes on the communication 
capabilities of the Internet by encouraging peer 
collaboration and contact with subject matter 
experts. 

15. Students are provided with a convenient means to 
interact with faculty and with other students taking 
the course or instructional program. 

16. Students are informed online about how to obtain 
help with coursework, exams, and assignments 

17. Online assessment methods are relevant and fair 
tests of achievement, based upon learning objectives 
and course performance expectations. 

18. Assessment methods emphasize authenticity in 
that performance is evaluated using practical 
(real-world) problem-solving tasks. 

19. Feedback on exams and assignments is 
constructive and timely. 



ERIC 



Boettcher & Conrad, 
1999 

Kerka, 1996 
McManus, 2001 
Palloff, 1999 
Sonwalker, 2001b 
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1. Students are given instructions and guidance on 
necessary website navigation tor the course. 

2. Web page navigation links are meaningfully labeled. 

3. Information is logically organized into related sets 
or chunks. 

4. Information sets or chunks are logically located on 
display screen. 

5. Information sets or chunks appear consistently in 
expected display locations. 

6. Consistency is maintained in information formats, 
content layout, and content location on the screen. 

7. Logical layout and spacing are used to control 
display density. 

8. Minimum font size is 9 points (10-12 points 
recommended). 

9. Upper- and lower-case letters are used for text. 

10. Font size is varied only for emphasis. 

1 1 . Use of blue and red colors for text is avoided. 

12. Web pages start and end on a coherent topic. 

13. Page change function, rather than scroll, is used 
at the end of a logical segment. 

14. Landmarks and signposts are used to show the 
user his or her present location in instructional 
sequence or website geography, path traversed, 
and what lies ahead. 

15. A visual sign is used to identify current operating 
mode, and initial changes in mode are indicated by 
auditory tone, if multimode functions are accessed. 
(Applies mainly to simulations and animated 
demonstrations.) 

16. A visual sign or selectable auditory signal, or both, 
are used to indicate the start and end of a task. 

17. The user is given a clearly indicated option to 
escape from a specific mode, operating condition, ^ 
unwanted website or page. For example, an option 
to page back or return to a home page position is 
always available and clearly indicated. 

18. Accommodations are made in display and auditory 
presentations for students with modest visual and 
hearing impairments (e.g., selectable font size and 
text-audio redundancy). 





Jones, 1989 

Jordan, 1988 

Nielsen, 2000 

Shneiderman, 

1998 
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Appendix D. Institutional and Support Infrastructure 

1 . The student is informed about the technology 
required for taking a course or instructional 
program. 

2. The student is informed about the technical 
competency required to take a course or a specific 
instructional program. 

3. The student is informed about the institutional 
requirements, including cost of course or program, 
course duration, time allowed for completion, 
any prerequisites or special skills required, and 
support services available. 

4. Instructional materials are kept current through 
timely updates. 

5. Available student services are clearly defined and 
accessible. 

6. The technology delivery system is adequate to 
support courses offered and student loads. 

7. The technology delivery system is as reliable and 
fail-safe as possible. 

8. The student is informed about how to obtain help 
with technology issues. 

9. The student is provided with necessary library 
and informational resources to meet course 
objectives. 

10. Convenient access is provided for students to 
obtain assistance with administrative and 
technology problems. 



20 
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